Linking Single System Synchronous Inter-Domain Transaction Activity

ABSTRACT

An approach is provided to correlate transaction data occurring at two different domains running on a common operating system image without using static, or common, correlators. Request-type event records are collected at a first domain within the operating system image, with each of the request-type event records including execution identifiers and a unique token that indicates the order in which the corresponding request-type event occurred on the first domain. Similarly, response-type event records are collected at a second domain within the operating system image. The request-type event records are matched with the response-type event records based on the execution identifiers and an overall order that is indicated by unique tokens included in the records. The matching of request-type event records with response-type event records indicate a number of inter-domain transactions which are recorded in a correlation data store.

TECHNICAL FIELD

The present disclosure relates to an approach that links synchronoustransaction activity that occurs in different domains. In particular,the present disclosure links the activity without use of staticcorrelators.

BACKGROUND OF THE INVENTION

Transaction tracking technologies focus on tracking compositeapplications across multiple technologies, protocols, domains(middleware stacks) and operating systems. Tracking is often achieved byinstrumenting targeted software with tracking agents which generatetracking events at strategic points in the application flow. Collectedtracking events can be analyzed to determine application metrics andtopology.

One of the challenges for transaction tracking is topology building.Making the necessary associations between requests from an applicationin one domain with the corresponding requests in the adjacent domain canbe difficult. For example, a first transaction process running on anoperating system may request a service from a second process, such as aqueue manager. Transaction tracking needs be able to match the requestfrom the first transaction process with the response from the secondprocess in order to track the interaction between the two processes.Traditional transaction tracking technologies generally employ acorrelator to make the association between corresponding inter-domaintransaction interactions. A correlator may be passed from the sourcedomain to the target domain (static correlator) or may be generatedindependently on each domain using common but unique data (dynamiccorrelator). In either case, matching correlators are used to associatetransaction interactions. A problem exists when it is undesirable or notpossible to pass a static correlator between domains and when commonunique data is not available to generate a dynamic correlator.

SUMMARY

An approach is provided to correlate transaction data occurring at twodifferent domains running on a common operating system image withoutusing static, or common, correlators. Request-type event records arecollected at a first domain within the operating system image, with eachof the request-type event records including execution identifiers and aunique token that indicates the order in which the correspondingrequest-type event occurred on the first domain. Similarly,response-type event records are collected at a second domain within theoperating system image, with each of the response-type event recordsalso including the execution identifiers and a unique token thatindicates the order in which the corresponding response-type eventoccurred. The request-type event records are matched with theresponse-type event records based on the execution identifiers and anoverall order that is indicated by unique tokens included in therecords. The matching of request-type event records with response-typeevent records indicate a number of inter-domain transactions which arerecorded in a correlation data store.

In a further embodiment, an approach is provided that intercepts aninter-domain event that occurs between a first domain and a seconddomain. The type of the inter-domain event is identified as being eithera “request” or a “response” event. Execution identifiers pertaining tothe inter-domain event are gathered, such as the system identifier, theprocess identifier, and the thread identifier. A unique token thatindicates an order that the inter-domain event occurred is generated,such as a timestamp-based token. The gathered execution identifiers,generated unique token, and the type of inter-domain event are stored ina data store for future transaction correlation processing with datagathered from another domain.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations, and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. Otheraspects, inventive features, and advantages of the present invention, asdefined solely by the claims, will become apparent in the non-limitingdetailed description set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings, wherein:

FIG. 1 is a block diagram of a data processing system in which themethods described herein can be implemented;

FIG. 2 provides an extension of the information handling systemenvironment shown in FIG. 1 to illustrate that the methods describedherein can be performed on a wide variety of information handlingsystems which operate in a networked environment;

FIG. 3 is a high-level diagram showing instrumentation collecting dataat two domains without use of common correlators;

FIG. 4 is a flowchart showing steps taken by processes running on twodomains within a common operating system and collecting instrumentationdata for future process correlation;

FIG. 5 is a flowchart showing steps performed by data collector processrunning on each domain; and

FIG. 6 is a flowchart showing steps performed by the correlation processto correlate the instrumentation data gathered during execution ofprocesses running on the domains of the operating system.

DETAILED DESCRIPTION

Certain specific details are set forth in the following description andfigures to provide a thorough understanding of various embodiments ofthe invention. Certain well-known details often associated withcomputing and software technology are not set forth in the followingdisclosure, however, to avoid unnecessarily obscuring the variousembodiments of the invention. Further, those of ordinary skill in therelevant art will understand that they can practice other embodiments ofthe invention without one or more of the details described below.Finally, while various methods are described with reference to steps andsequences in the following disclosure, the description as such is forproviding a clear implementation of embodiments of the invention, andthe steps and sequences of steps should not be taken as required topractice this invention. Instead, the following is intended to provide adetailed description of an example of the invention and should not betaken to be limiting of the invention itself. Rather, any number ofvariations may fall within the scope of the invention, which is definedby the claims that follow the description.

The following detailed description will generally follow the summary ofthe invention, as set forth above, further explaining and expanding thedefinitions of the various aspects and embodiments of the invention asnecessary. To this end, this detailed description first sets forth acomputing environment in FIG. 1 that is suitable to implement thesoftware and/or hardware techniques associated with the invention. Anetworked environment is illustrated in FIG. 2 as an extension of thebasic computing environment, to emphasize that modern computingtechniques can be performed across multiple discrete devices.

FIG. 1 illustrates information handling system 100, which is asimplified example of a computer system capable of performing thecomputing operations described herein. Information handling system 100includes one or more processors 110 coupled to processor interface bus112. Processor interface bus 112 connects processors 110 to Northbridge115, which is also known as the Memory Controller Hub (MCH). Northbridge115 connects to system memory 120 and provides a means for processor(s)110 to access the system memory. Graphics controller 125 also connectsto Northbridge 115. In one embodiment, PCI Express bus 118 connectsNorthbridge 115 to graphics controller 125. Graphics controller 125connects to display device 130, such as a computer monitor.

Northbridge 115 and Southbridge 135 connect to each other using bus 119.In one embodiment, the bus is a Direct Media Interface (DMI) bus thattransfers data at high speeds in each direction between Northbridge 115and Southbridge 135. In another embodiment, a Peripheral ComponentInterconnect (PCI) bus connects the Northbridge and the Southbridge.Southbridge 135, also known as the I/O Controller Hub (ICH) is a chipthat generally implements capabilities that operate at slower speedsthan the capabilities provided by the Northbridge. Southbridge 135typically provides various busses used to connect various components.These busses include, for example, PCI and PCI Express busses, an ISAbus, a System Management Bus (SMBus or SMB), and/or a Low Pin Count(LPC) bus. The LPC bus often connects low-bandwidth devices, such asboot ROM 196 and “legacy” I/O devices (using a “super I/O” chip). The“legacy” I/O devices (198) can include, for example, serial and parallelports, keyboard, mouse, and/or a floppy disk controller. The LPC busalso connects Southbridge 135 to Trusted Platform Module (TPM) 195.Other components often included in Southbridge 135 include a DirectMemory Access (DMA) controller, a Programmable Interrupt Controller(PIC), and a storage device controller, which connects Southbridge 135to nonvolatile storage device 185, such as a hard disk drive, using bus184.

ExpressCard 155 is a slot that connects hot-pluggable devices to theinformation handling system. ExpressCard 155 supports both PCI Expressand USB connectivity as it connects to Southbridge 135 using both theUniversal Serial Bus (USB) the PCI Express bus. Southbridge 135 includesUSB Controller 140 that provides USB connectivity to devices thatconnect to the USB. These devices include webcam (camera) 150, infrared(IR) receiver 148, keyboard and trackpad 144, and Bluetooth device 146,which provides for wireless personal area networks (PANs). USBController 140 also provides USB connectivity to other miscellaneous USBconnected devices 142, such as a mouse, removable nonvolatile storagedevice 145, modems, network cards, ISDN connectors, fax, printers, USBhubs, and many other types of USB connected devices. While removablenonvolatile storage device 145 is shown as a USB-connected device,removable nonvolatile storage device 145 could be connected using adifferent interface, such as a Firewire interface, etcetera.

Wireless Local Area Network (LAN) device 175 connects to Southbridge 135via the PCI or PCI Express bus 172. LAN device 175 typically implementsone of the IEEE 0.802.11 standards of over-the-air modulation techniquesthat all use the same protocol to wireless communicate betweeninformation handling system 100 and another computer system or device.Optical storage device 190 connects to Southbridge 135 using Serial ATA(SATA) bus 188. Serial ATA adapters and devices communicate over ahigh-speed serial link. The Serial ATA bus also connects Southbridge 135to other forms of storage devices, such as hard disk drives. Audiocircuitry 160, such as a sound card, connects to Southbridge 135 via bus158. Audio circuitry 160 also provides functionality such as audioline-in and optical digital audio in port 162, optical digital outputand headphone jack 164, internal speakers 166, and internal microphone168. Ethernet controller 170 connects to Southbridge 135 using a bus,such as the PCI or PCI Express bus. Ethernet controller 170 connectsinformation handling system 100 to a computer network, such as a LocalArea Network (LAN), the Internet, and other public and private computernetworks.

While FIG. 1 shows one information handling system, an informationhandling system may take many forms. For example, an informationhandling system may take the form of a desktop, server, portable,laptop, notebook, or other form factor computer or data processingsystem. In addition, an information handling system may take other formfactors such as a personal digital assistant (PDA), a gaming device, ATMmachine, a portable telephone device, a communication device or otherdevices that include a processor and memory.

The Trusted Platform Module (TPM 195) shown in FIG. 1 and describedherein to provide security functions is but one example of a hardwaresecurity module (HSM). Therefore, the TPM described and claimed hereinincludes any type of HSM including, but not limited to, hardwaresecurity devices that conform to the Trusted Computing Groups (TCG)standard, and entitled “Trusted Platform Module (TPM) SpecificationVersion 1.2.” The TPM is a hardware security subsystem that may beincorporated into any number of information handling systems, such asthose outlined in FIG. 2.

FIG. 2 provides an extension of the information handling systemenvironment shown in FIG. 1 to illustrate that the methods describedherein can be performed on a wide variety of information handlingsystems that operate in a networked environment. Types of informationhandling systems range from small handheld devices, such as handheldcomputer/mobile telephone 210 to large mainframe systems, such asmainframe computer 270. Examples of handheld computer 210 includepersonal digital assistants (PDAs), personal entertainment devices, suchas MP3 players, portable televisions, and compact disc players. Otherexamples of information handling systems include pen, or tablet,computer 220, laptop, or notebook, computer 230, workstation 240,personal computer system 250, and server 260. Other types of informationhandling systems that are not individually shown in FIG. 2 arerepresented by information handling system 280. As shown, the variousinformation handling systems can be networked together using computernetwork 200. Types of computer network that can be used to interconnectthe various information handling systems include Local Area Networks(LANs), Wireless Local Area Networks (WLANs), the Internet, the PublicSwitched Telephone Network (PSTN), other wireless networks, and anyother network topology that can be used to interconnect the informationhandling systems. Many of the information handling systems includenonvolatile data stores, such as hard drives and/or nonvolatile memory.Some of the information handling systems shown in FIG. 2 depictsseparate nonvolatile data stores (server 260 utilizes nonvolatile datastore 265, mainframe computer 270 utilizes nonvolatile data store 275,and information handling system 280 utilizes nonvolatile data store285). The nonvolatile data store can be a component that is external tothe various information handling systems or can be internal to one ofthe information handling systems. In addition, removable nonvolatilestorage device 145 can be shared among two or more information handlingsystems using various techniques, such as connecting the removablenonvolatile storage device 145 to a USB port or other connector of theinformation handling systems.

FIG. 3 is a high-level diagram showing instrumentation collecting dataat two domains without use of common correlators. Common operatingsystem image 300 executes a number of domains, such as processes,subsystems, applications, and the like. Two such domains are shown beingexecuted by operating system image 300 in FIG. 3. These domains, Domain“A” (310) and Domain “B” (320) are each executing within commonoperating system image 300 with one, or both, of the domains providing aservice, or functionality, to the other domain (e.g., Domain “A” (310)“puts” a request to Domain “B” (320) via an Application ProgrammingInterface (“API”) call, etc.). By running in a common operating systemimage, each of the domains has common execution identifiers with theother domain. These execution identifiers can include a systemidentifier, a process identifier, a thread identifier, etc.

When tracking transactions through the system, instrumentation isenabled in each of the domains. This instrumentation acts to interceptinter-domain events that occur between the domains (e.g., request eventsand response events, etc.). In addition, the instrumentation generates aunique token at each domain, such as a timestamp-based token or anincremental token such as an integer counter, etc., that can be used todetermine the order in which events occurred. In one embodiment, one ofthe domains (e.g., Domain “A”) is the “requestor” that generatesrequest-type events when requesting services from the other domain andthe other domain (e.g., Domain “B”) is the responder that provides theservice and generates response-type events. Request-type events generaterequest-type event records by the instrumentation. The request-typeevent records include the execution identifiers (e.g., a systemidentifier, a process identifier, a thread identifier, etc.), the typeof event (in this case indicating that it is a “request” event), and theunique token that was generated by the requesting domain. Likewise, theresponding domain provides the service and generates response-type eventrecords. The response-type event records include the same executionidentifiers (e.g., a system identifier, a process identifier, a threadidentifier, etc.) as the counterpart response-type event record.However, the type of event now indicates that it is a “response” event,and the unique token is generated at the response domain and is notbased on the requestor's unique token per se, however the scheme oralgorithm used to generate the responder's unique token can be the sameas the scheme or algorithm that was used to generate the requestor'sunique token (e.g., a timestamp-based unique token, etc.). Therequest-type event records generated by the instrumentation running onDomain “A” are stored in data store 330 (e.g., a memory, nonvolatilestorage, etc.) which is local to Domain “A”. Similarly, theresponse-type event records generated by the instrumentation running onDomain “B” are stored in data store 340 which is local to Domain “B”.

Correlation process 350 is performed to match request-type event records(e.g., generated by Domain “A”) with their respective response-typeevent records (e.g., generated by Domain “B”) without the use of anystatic, or common, correlators or tokens. Instead, the request-typeevent records are merged with the response-type event records and sortedusing the unique tokens that were generated by the instrumentation. Thecorrelation process selects request-type event records from the mergeddata and then searches for a matching response-type event record (arecord with the same execution identifiers) and ensures that there areno intervening request-type event records with the same executionidentifiers. When a request-type event record is matched with aresponse-type event record, the matched pair indicates an inter-domaintransaction that is written to correlated transaction data store 360.The correlation process continues to select request-type event recordsand find their matching response-type event records for as many otherrecords in the merged data that are desired for processing. Theresulting inter-domain transaction data are written to correlatedtransaction data store 360 for eventual analysis.

After the inter-domain transactions have been identified, transactiontracking software 370 analyzes the inter-domain transactions stored indata store 360 and generates one or more analyses of the inter-domaintransactions (analysis 380). System administrators and developers canthen review the analysis of inter-domain transactions in order toidentify bottlenecks or other areas of improvement in either of thedomains.

FIG. 4 is a flowchart showing steps taken by processes running on twodomains within a common operating system and collecting instrumentationdata for future process correlation. In the example shown in FIG. 4,Domain “A” (400) is running a process, such as Customer InformationControl System™ (CICS) distributed by International Business MachinesCorporation, and Domain “B” (450) is running another process, such asWebSphere Messaging Queue™ (WMQ) also distributed International BusinessMachines Corporation.

Domain “A” has a data collector that collects process information wheninstrumentation is enabled. Likewise, Domain “B” also has a datacollector that collects process information when instrumentation isenabled. Processing by Domain “A” data collector commences at 410whereupon, at step 415, the domain runs a process (such as CICS™) withinstrumentation enabled. At predefined process 420, the instrumentationsoftware intercepts inter-domain events (e.g., API calls, etc.) andcollects relevant process data (see FIG. 5 and corresponding text forprocessing details). The collected data (e.g., request-type eventrecords, etc.) are stored in local data store 425 which is a memory arealocal to Domain “A.” A decision is made as to whether processing hasended or terminated (decision 430). If processing has not ended, thendecision 430 branches to the “no” branch which loops back to continuerunning the process (e.g., CICS™, etc.) and continue collecting therelevant process data. This looping continues until process ends, atwhich point decision 430 branches to the “yes” branch and processing bythe data collector running on Domain “A” ends at 435.

Turning now to response-type event handling, processing by Domain “B”data collector commences at 460 whereupon, at step 465, the domain runsa process (such as WMQ™) with instrumentation enabled. At predefinedprocess 470, the instrumentation software intercepts inter-domain events(e.g., response events resulting from WMQ handling the request fromCICS, etc.) and collects relevant process data (see FIG. 5 andcorresponding text for processing details). The collected data (e.g.,response-type event records, etc.) are stored in local data store 475which is a memory area local to Domain “B.” A decision is made as towhether processing has ended or terminated (decision 480). If processinghas not ended, then decision 480 branches to the “no” branch which loopsback to continue running the process (e.g., WMQ™, etc.) and continuecollecting the relevant process data. This looping continues untilprocess ends, at which point decision 480 branches to the “yes” branchand processing by the data collector running on Domain “B” ends at 485.

FIG. 5 is a flowchart showing steps performed by data collector processrunning on each domain. Data collection processing commences at 500. Asshown, in one embodiment the data collection processing runs on eachdomain where instrumentation has been enabled. At step 510, the datacollector intercepts an inter-domain event such as a request (call) toanother domain from this domain or a response to another domain by thisdomain. At step 520, the type of event is identified (e.g., arequest-type event, a response-type event, etc.). At step 530, datapertaining to the inter-domain event is gathered with this data formingan execution identifier. In one embodiment, the execution identifier isformed from one or more identifiers such as the system identifier, theprocess identifier, and the thread identifier. For a given transaction,the execution identifier is the same on both the requesting domain aswell as the responding domain. At step 540, a unique token is generatedwith the token indicating an order in which inter-domain events occurred(e.g., timestamp-based token, incremented integer based token, etc.). Atstep 550, the gathered request data that forms the execution identifieris stored along with the type of the event (request or response) as wellas the generated unique token. This data is stored in local data store560, such as a memory area, that is local to the domain that is runningthe data collector process.

A decision is made as to whether data collector processing is beingterminated on this domain (decision 570). If data collector processingis not being terminated on this domain, then decision 570 branches tothe “yes” branch which loop back to continue intercepting inter-domainrequests and responses and storing the relevant data as described above.This looping continues until data collector processing is beingterminated on this domain, at which point decision 570 branches to the“yes” branch whereupon data collector processing ends at 595.

FIG. 6 is a flowchart showing steps performed by the correlation processto correlate the instrumentation data gathered during execution ofprocesses running on the domains of the operating system. Correlationprocess commences at 600 whereupon, at step 610 the local data storesare sorted based on the unique tokens included in the event records ifthey are not already sorted (e.g., first local data store 425 is sortedby the unique tokens resulting in sorted first local data store 615 andsecond local data store 475 is sorted by the unique tokens resulting insorted second local data store 620). At step 625, the sorted eventrecords are merged with the resulting merge file (630) also being sortedbased on the unique token included in each event record. Resulting mergefile 630 is then processed to identify the inter-domain transactions asdiscussed below.

At step 640, the first request-type event record in merge file 630 isselected. In one embodiment, an indicator was inserted in each eventrecord indicating whether the event corresponded to a request or aresponse. A decision is made as to whether an request-type event recordwas found in merge file 630 (decision 650). If a request-type eventrecord was found in merge file 630, then decision 650 branches to the“yes” branch whereupon, at step 660, the process searches for a matchingresponse-type event record (e.g., a response-type record with the sameexecution identifier). In addition, at step 660, the correlation processensures that there were not any addition (intervening) request-typeevents with the same execution identifier that occurred before theresponse-type record was found. A decision is made by the correlationprocess as to whether a valid response-type event record was foundbefore any intervening request-type event records were encountered(decision 670). If a valid response-type event record was found, thenthe request-type event record and the matched response-type event recordindicate an inter-domain transaction. In this case, decision 670branches to the “yes” branch whereupon, at step 675, the inter-domaintransaction data is written to correlated inter-domain transactions datastore 360. Inter-domain transaction data can include the executionidentifiers as well as timing data that indicates when the request wasmade by the requesting domain as well as when the response was sent bythe responding domain. Processing then loops back to step 640 to selectthe next request-type event record from merged data store 630.

Returning to decision 670, if a valid response-type record was not found(e.g., an intervening request-type event record was encountered, noresponse was found, etc.), then decision 670 branches to the “no” branchwhereupon, at step 680, a record of the error can be written to errorlog data store 685 to indicate the request-type event record for whichno valid response-type event record was found. Processing then loopsback to step 640 to select the next request-type event record frommerged data store 630.

At step 640, the next request-type event record is selected from mergefile data store 630 and the processing described above searches for amatching response-type event record. This looping continues until allrequest-type event records have been processed, at which point decision650 branches to the “no” branch and processing ends at 695.

One of the preferred implementations of the invention is a clientapplication, namely, a set of instructions (program code) or otherfunctional descriptive material in a code module that may, for example,be resident in the random access memory of the computer. Until requiredby the computer, the set of instructions may be stored in anothercomputer memory, for example, in a hard disk drive, or in a removablememory such as an optical disk (for eventual use in a CD ROM) or floppydisk (for eventual use in a floppy disk drive). Thus, the presentinvention may be implemented as a computer program product for use in acomputer. In addition, although the various methods described areconveniently implemented in a general purpose computer selectivelyactivated or reconfigured by software, one of ordinary skill in the artwould also recognize that such methods may be carried out in hardware,in firmware, or in more specialized apparatus constructed to perform therequired method steps. Functional descriptive material is informationthat imparts functionality to a machine. Functional descriptive materialincludes, but is not limited to, computer programs, instructions, rules,facts, definitions of computable functions, objects, and datastructures.

While particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein, that changes and modifications may bemade without departing from this invention and its broader aspects.Therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this invention. Furthermore, it is to be understood that theinvention is solely defined by the appended claims. It will beunderstood by those with skill in the art that if a specific number ofan introduced claim element is intended, such intent will be explicitlyrecited in the claim, and in the absence of such recitation no suchlimitation is present. For non-limiting example, as an aid tounderstanding, the following appended claims contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimelements. However, the use of such phrases should not be construed toimply that the introduction of a claim element by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim element to inventions containing only one such element,even when the same claim includes the introductory phrases “one or more”or “at least one” and indefinite articles such as “a” or “an”; the sameholds true for the use in the claims of definite articles.

What is claimed is:
 1. An information handling system comprising: one ormore processors; a memory coupled to at least one of the processors; anoperating system image stored in the memory and executed by at least oneof the processors; a set of instructions stored in the memory andexecuted by at least one of the processors, wherein the set ofinstructions perform actions of: intercepting an inter-domain eventbetween a first domain and a second domain, wherein the first and seconddomains are running within the operating system image, and wherein afirst data collector in the first domain generates correspondingexecution identifiers for events originating within the first domain anda second data collector in the second domain generates correspondingexecution identifiers for events originating within the second domain;identifying a type of the inter-domain event; gathering one or moreselected execution identifiers pertaining to the inter-domain event,wherein the selected execution identifiers include a system identifier,a process identifier, and a thread identifier; generating a unique tokenthat indicates an order that the inter-domain event occurred whencompared with a plurality of unique tokens corresponding to otherinter-domain events; and storing the gathered selected executionidentifiers, the generated unique token, and the type of inter-domainevent in a data store.
 2. The information handling system of claim 1wherein the intercepting, identifying, gathering, generating, andstoring are included in an instrumentation routine and are performedwhen instrumentation is enabled.
 3. The information handling system ofclaim 1 wherein the generated unique token is based on a timestamp.
 4. Acomputer program product comprising a non-transitory computer readablemedium with functional descriptive material stored thereon, that, whenexecuted by an information handling system, causes the informationhandling system to perform actions that include: intercepting aninter-domain event between a first domain and a second domain, whereinthe first and second domains are running within a common operatingsystem image, and wherein a first data collector in the first domaingenerates corresponding execution identifiers for events originatingwithin the first domain and a second data collector in the second domaingenerates corresponding execution identifiers for events originatingwithin the second domain; identifying a type of the inter-domain event;gathering one or more selected execution identifiers pertaining to theinter-domain event, wherein the selected execution identifiers include asystem identifier, a process identifier, and a thread identifier;generating a unique token that indicates an order that the inter-domainevent occurred when compared with a plurality of unique tokenscorresponding to other inter-domain events; and storing the gatheredselected execution identifiers, the generated unique token, and the typeof inter-domain event in a data store.
 5. The computer program productof claim 4 wherein the intercepting, identifying, gathering, generating,and storing are included in an instrumentation routine and are performedwhen instrumentation is enabled.
 6. The computer program product ofclaim 4 wherein the generated unique token is based on a timestamp.